Interactivity via mobile image recognition

ABSTRACT

Systems and methods of interacting with a virtual space, in which a mobile device is used to electronically capture image data of a real-world object, the image data is used to identify information related to the real-world object, and the information is used to interact with software to control at least one of: (a) an aspect of an electronic game; and (b) a second device local to the mobile device. Contemplated systems and methods can be used to gaming, in which the image data can be used to identify a name of the real-world object, to classify the real-world object, identify the real-world object as a player in the game, to identify the real-world object as a goal object or as having some other value in the game, to use the image data to identify the real-world object as a goal object in the game.

This application is a divisional of Ser. No. 13/069,157, filed Mar, 22,2011, which is a divisional of Ser. No. 11/510,009 filed Aug. 25, 2006and issued as U.S. Pat. No. 8,130,242 on Mar. 6, 2012, which claims thebenefit of U.S. provisional application No. 60/712,590, filed Aug. 29,2005, and is a continuation-in-part of Ser. No. 11/294,971, filed Dec.5, 2005 and issued as U.S. Pat. No. 7,403,652 on Jul. 22, 2008, which isa continuation of Ser. No. 09/992,942, filed Nov. 5, 2001 and issued asU.S. Pat. No. 7,016,532 on Mar. 21, 2006, which claims priority to U.S.provisional application No. 60/317,521, filed Sep. 5, 2001 and U.S.provisional application No. 60/246,295, filed Nov. 6, 2000. All of theabove-listed applications are incorporated herein by reference.

FIELD OF THE INVENTION

The invention pertains to the field of mobile networks, mobile devicessuch as telephones, and information provided to and from users throughsuch devices.

BACKGROUND OF THE INVENTION

U.S. Pat. No. 7,016,532 to Boncyk et al., issued Mar. 21, 2006, andincorporated herein by reference in its entirety, describes a method andprocess through which individuals can use their cell phones, PDAs andthe like to take digital images of two and three dimensional objects,the image(s) or information derived from the image(s) can be sent to adistal server, and the server can use the transmitted information toidentify an object within the image. Having identified the object theserver can then provide additional information (telephone number,address, web links, and so forth) back to the individual taking thedigital image. That person, in turn, can use the additional informationin any suitable manner, such as to execute a purchase, surf theInternet, and so forth.

It is also known to use one's phone to interact in limited ways with avirtual game world.

For example, a cell phone can be used as a golf club to interact with avirtual golf course.http://mobhappy.typepad.com/russell_buckleys_mobhappy/2005/01/index.html.As another example, a cell phone can be used to play a virtual treasurehunt, http://www.joystiq.com/ 2006/02/24/gps-gaming/, and to leave orfind virtualgraffiti,http://www.dw-world.de/dw/article/0,15641481993,00.html.

What has not been appreciated, however, is that a camera enabled mobiledevice can be used in concert with software to identify informationrelated to real-world objects, and then use that information to controleither (a) an aspect of an electronic game, or (b) a second device localto the mobile device.

SUMMARY OF THE INVENTION

The present invention provides systems, methods, and apparatus in whicha camera enabled mobile device is used in concert with software toidentify information related to real-world objects, and then use thatinformation to control either (a) an aspect of an electronic game, or(b) a second device local to the mobile device.

In contemplated uses, the other inputs can be almost anything, includingfor example, a password, use of a button as a trigger of a pretendweapon, checking off steps in a treasure hunt, playing a video game thathas both real-world and virtual objects, voting, and so forth.

The combination of real world situation and virtual world situation canalso be almost anything. For example, the real world situation can varyfrom relatively static (such as an advertisement in a magazine) torelatively dynamic (such as cloud formations, images on a televisionset, location of a person or automobile). Moreover, the virtual worldsituation can independently vary from relatively static (such as anoption to purchase virtual money or other resources) to relativelydynamic (such as the positions of virtual characters in a video game).

Preferred embodiments of the inventive subject matter of thisapplication include the following steps. Steps 1 and 2 of this processwere disclosed in U.S. Pat. No. 7,016,532.

1) An information connection is established between a mobile device andan information resource (such as a web site) based on imagery capturedby the mobile device. This is done by capturing an image of an objectwith the mobile device, sending the image to a distal server,recognizing the object in the server, and the server sending aninformation resource address to the mobile device.

2) The user obtains information from the information resource via themobile device.

3) The user interacts with the information resources or object based onthe previously established information connection. This interaction maybe of various types, including for example:

-   -   Repeating the above process multiple times.    -   Performing a transaction.    -   Performing actions in a game.    -   Opening a door (physical or virtual) to gain access to secure        information or a secure location.    -   Interacting with TV programming (including selecting a channel).    -   Communicating with other people.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 is a schematic of an exemplary method according to one aspect ofthe inventive subject matter.

FIG. 2 is a schematic of an exemplary method according to another aspectof the inventive subject matter.

DETAILED DESCRIPTION

Definitions

As used herein, the term “mobile device” means a portable device thatincludes image capture functionality, such as a digital camera, and hasconnectivity to at least one network such as a cellular telephonenetwork and/or the Internet. The mobile device may be a mobile telephone(cellular or otherwise), PDA, or other portable device.

As used herein, the term “application” means machine-executablealgorithms, usually in software, resident in the server, the mobiledevice, or both.

As used herein, the term “user” means a human being that interacts withan application.

As used herein, the term “server” means a device with at least partialcapability to recognize objects in images or in information derived fromimages.

In FIG. 1, a first exemplary class of processes 100 includes: step 110wherein a user captures at least one image of an object using a mobiledevice; step 120 wherein at least part of the image, or informationderived therefrom, or both, is sent via a network to a distal server;step 130 wherein the server recognizes at least one object in the image;and step 140 wherein the server determines some information, based onthe identity of the object and other information, such as the currenttime, the observed state of the object, the location of the user, etc.If the appearance of the object varies with time, then this time-varyingappearance may be used in determination of the information. Thistime-varying appearance may furthermore be correlated with the currenttime in determining the information.

Other contemplated steps include step 152 of providing information tothe user via a network and the mobile device; step 154 of sending aninformation address to the user via a network and the mobile device;step 156 of sending an instruction to a computer, machine, or otherdevice to perform an action; and step 158 of the user performing anaction based on the action performed by the application.

The above process may be repeated as many times as is desired orappropriate. The user may capture at least one additional image orprovide other inputs to the server or to another device, based on theaction performed by the application, thus beginning a new cycle.

In FIG. 2, another class of methods 200 of interacting with a virtualspace, comprises: step 210 of using a mobile device to electronicallycapture image data of a real-world object; step 220 of using the imagedata to identify information related to the real-world object; and step230 of using the information to interact with software being operated atleast in part externally to the mobile device, to control at least oneof: (a) an aspect of an electronic game; and (b) a second device localto the mobile device.

Option steps collectively shown as 242 include using the mobile deviceto electronically capture a still video or a moving image.

Optional steps collectively shown as 244 include using the image data toidentify a name of the real-world object, to classify the real-worldobject, identify the real-world object as a player in the game, toidentify the real-world object as a goal object or as having some othervalue in the game, to use the image data to identify the real-worldobject as a goal object in the game, or to ascertain an environmentalcharacteristic nearby the mobile device.

Optional steps collectively shown as 246 include the softwareaccommodating at least three or more preferably at least five concurrentusers who may interact with another of the users.

Optional steps collectively shown as 248 comprise providing an input tothe game, such as data relating to use of a virtual weapon, virtualplaying of music, or virtual traveling.

Optional steps collectively shown as 250 comprise changing a channel, orin some other manner controlling a TV or other device.

Optional steps collectively shown as 252 further comprise using adesignator of physical location of the mobile device to interact withthe software, including for the designator location comprises ageographic coordinate.

Optional steps collectively shown as 254 further comprise using at leastone of orientation and acceleration of the mobile device to interactwith the software.

EXAMPLES

In FIG. 1, a system includes a user who uses a cell phone or othermobile device to capture an image of an object . All practical objectsare contemplated, including for example a cover of a CD (compact audiodisk) or a visible image on a face of the CD, a DVD (digital videodisk), a magazine advertisement, a consumer product, and so forth.Identification of the object is added to the user's online “shoppingcart” in an online shopping application. The shopping cart represents alist of items that the user intends to purchase. The user then continuesto shop by capturing images of additional objects that he either intendsto purchase or about which he desires information.

A user deduces, from information in a game application, the identity,nature, and/or location of a “goal object” that he should find as a stepin a game. The user then finds a “candidate object” that he believes tobe either the goal object or another object that is either nearby thegoal object or on the path to the goal object, or is otherwise relatedto his search for the goal object. The user captures an image of thecandidate object with his cell phone. The image is sent to the serverand recognized. If the candidate object is the goal object, the userobtains points in the game. If the candidate object is not the goalobject but instead is on the path to or nearby the goal object, then theapplication may provide the user with A) information regarding hisprogress towards the goal object and/or B) a hint regarding how toprogress towards the goal object. goal objects, reward points, hints,and various other aspects of such a game may be dynamic, so that thegame changes with time, location, participants, participants' states andprogress, and other factors.

A user captures an image of a building, store, statue, or other such“target object.” Interactive content and/or information pertinent to thetarget object is provided to the user via the mobile device. Theinteractive content and/or information is created and/or modified basedon the appearance of the target object. For example, advertisements forcold refreshments may be sent to the user based on the determining thatthe weather at the user's location is hot and sunny. Such determinationof conditions at the user's location may be based on at least one of: A)the appearance of shadows in the image, B) temperature data obtainedfrom weather information resources, C) the location of the mobile deviceas determined by Global Positioning System, radio frequency rangingand/or triangulation, or other means, D) the appearance of lights (e.g.street lights, neon signs, illuminated billboards, etc.), and E) currenttime.

A user wishes to gain access to a secure location, information resource,computing resource, or other such thing (the “secure resource”) that isrestricted from general public access. The user captures an image, withhis mobile device, of the secure resource or an object, such as a sign,that is nearby or otherwise corresponds to the secure resource. Theimage is sent to a server. The server determines that the user wishes togain access to the secure resource. The server sends a message to theuser (via the mobile device), instructing the user to provide an imageof the user's face and/or some other identifying thing. The user thencaptures an image of his face or other identifying thing and this imageis sent to the server. The server validates the identity of the user byrecognizing the user's face or other identifying thing in the image. Theserver then instructs the user to provide a password. The user providesthe password, by speaking it into the mobile device, entering it into akeyboard on the mobile device, or entering it into a keyboard on anotherdevice (such as a keyboard attached to the secure resource), or othermeans. The password may vary depending on the secure resource, theidentity of the user, the current time, and other factors. The server oranother device then grants or denies the user access to the secureresource based on verification of the password, current time, useridentity, user location, secure resource location, and/or other factors.

A game involving simulated shooting of a weapon may be provided asfollows. A user points his mobile device at an object that he wishes toshoot. The user sees, in the screen display of his mobile device, asimulated view of using a weapon. For example, the user may see thecrosshairs of an aiming sight superimposed on the real-world scene infront of him. The user “shoots” a simulated weapon by pressing a buttonor making some other input (e.g. screen input or voice command) to themobile device. The mobile device captures an image and sends it to theserver. Other information may also be sent to the server in addition tothe image. The application (comprising software on one or both of theserver and mobile device) recognizes the object(s) in the image andcorrelates them to the simulated weapon aim point. The application thenprovides a simulation, on the mobile device screen, of the weaponfiring. This simulation may be superimposed on the image of thereal-world scene. Depending on various factors, the weapon may havevarious effects within the game, from no effect at all to completelydestroying a simulated target. Such effects may be simulated viaanimation, video, and/or audio in the mobile device. Such effects may begenerated in the server, mobile device, or both, or downloaded from theserver or another computer. The result of the shooting the weapon maydepend on various factors, including the identity of the objects in theimage and the position of those objects relative to the user andrelative to the weapon aimpoint.

Multiple users may simulate fighting against each other. In such a case,if a user shoots another user, then the mobile devices of each playerwould display appropriate outputs. For example, if one user (the“Victim”) is shot by another, then the Victim's mobile device mayproduce animations and sound effects portraying the attack from thereceiving side. The Victim may be have points (score, health, orotherwise) deducted from his game points due to such an attack. userswithin such a game, and their positions relative to other users andweapon aim points, may be determined via various means. Such means mayinclude, for example, “bulls-eye” tags worn by users. In this case, forexample, a Victim might only be successfully “shot” if bulls-eye symbolappears in the part of the image that corresponds the weapon aim point.

Other simulated weapons, such as swords, shields, missiles, projectiles,or beam weapons may also be used in such a game.

If orientation, acceleration, and/or positions sensor are included inthe mobile device, then the orientation and/or acceleration of themobile device may be used as inputs to an application such as a game.For example, a user may engage in simulated sword fighting bycontrolling his simulated sword through movement of his mobile device.Additional examples are flying, driving, or other simulators in whichthe user controls a simulated object via motion of his mobile device. Insuch games, the game may be displayed by the mobile device or some otherdevice, such as a television or computer. In this case, the mobiledevice serves, in essence, as a mouse, joystick, drawing pen, or othermanual input device to a computing system. The orientation and/oracceleration sensors may be internal to the mobile device or may beimplemented completely or partially external to the mobile device (forexample, using radio-frequency or magnetic position determination).

A user may use his mobile device to interact with content, where“content” means electronically provided programming, games, or otherinformation. Example of content in this context are: televisionprograms, computer games, video games, radio programs, motion pictures,music, news programs, etc. In this application, the user captures animage of at least one object, an object in the image is recognized by aserver, and then based on the identity of the object, and optionallyalso the identity of the user, the current time, and other such factors,the content is modified.

An example of such usage is a user capturing an image of anadvertisement or other item in a magazine or newspaper and thus causinghis television to receive content appropriate to the item. This may beaccomplished by the server sending a message A) to the user'stelevision, instructing the television to change the channel or B) toanother server or computing system that in turn sends content to theuser's television. This process may be accomplished not only throughtelevision but also through any device capable of providing content tothe user, including for example, a computer, a radio, an audio device,or a game device.

After the user has initiated reception of the content, he may continueto interact with the content via capture of further images, motion ofthe mobile device, or other inputs. For example, a user may capture animage of an electronic billboard (or other electronic display). Theserver recognizes the image on the billboard and then establishes acommunication path between the user and the computer that controls thebillboard. The billboard may then display new and interactive content tothe user, including visual and audio content. The user may interact withthis content, via the billboard, through further image capture and/ormotion of the mobile device. The content in such interaction may beprovided to the user through the billboard, the mobile device, or anycombination of thereof. Such interaction may be used for advertising(e.g. via a billboard), entertainment (e.g. via a computer, television,or other such device with audio and/or video display capability), work,study, etc. Such interaction may also be used for interactive machines,such as vending machines, ticket machines, information kiosks, etc.

Multiple users can interact with each other. users can be connectedtogether in a virtual space, community, or environment by having“linked” to content based on “starting points” (real world physicalobjects) that are in some way related.

For example, several users could link to each other, by capturing imagesof the same billboard (interactive or otherwise). These users could thenparticipate in the same interactive experience that is being displayedon the billboard and/or on their mobile devices. These users wouldgenerally be in physical proximity to each other. An example would bethe spectators at a sports event interacting with the event via theirmobile devices by having “clicked” (captured images) of the scoreboardor other display. Another example is multiple users in front of the samedynamic display (e.g. large screen display) and interacting with boththe display content and each other. users at a meeting or convention cancast votes or otherwise interact with the group and other users.

Users may similarly participate in a common virtual environment eventhough they are not physically close to each other. An example would bemultiple users “clicking” on (capturing images of) the same type ofbeverage bottle and thus being connected together. Another example wouldbe multiple users “clicking” on a television program or Internet-basedprogram and similarly being connected together. users at meetings caninteract with other users that might not be in physical attendance butare attending via electronic connection. Remote attendees (notphysically present) of such a meeting can also interact with the meetingin general.

Users may interact directly with television or other such audio/videocontent. This is accomplished by capturing an image of an object,recognizing the object in a server, and then connecting the user to acomputing system that interacts with both the user and the content. Forexample, users may “click” on (capture an image of) the image of atelevision program on their television screen. Based on recognition ofwhat is on the screen, they are then connected to a computing systemthat interacts with the television program. In this manner, the userscan interact with the television program by, for example, voting forparticipants, voting for or otherwise selecting the next steps in astory or the desired outcome, playing the role of a character in astory, etc. This technique may be applied to not only television, butalso any other form of electronically provided entertainment, such asdigital motion pictures, and computer games.

Thus, specific embodiments and applications have been disclosed in whicha camera enabled mobile device is used in concert with software toidentify information related to real-world objects, and then use thatinformation to control either (a) an aspect of an electronic game, or(b) a second device local to the mobile device. It should be apparent,however, to those skilled in the art that many more modificationsbesides those already described are possible without departing from theinventive concepts herein. The inventive subject matter, therefore, isnot to be restricted except in the spirit of the appended claims.Moreover, in interpreting both the specification and the claims, allterms should be interpreted in the broadest possible manner consistentwith the context. In particular, the terms “comprises” and “comprising”should be interpreted as referring to elements, components, or steps ina non-exclusive manner, indicating that the referenced elements,components, or steps may be present, or utilized, or combined with otherelements, components, or steps that are not expressly referenced. Wherethe specification claims refers to at least one of something selectedfrom the group consisting of A, B, C . . . and N, the text should beinterpreted as requiring only one element from the group, not A plus N,or B plus N, etc.

What is claimed is:
 1. A method of assessing environmentalcharacteristics, the method comprising: capturing, via a mobile device,video representing a surrounding environment; analyzing, the surroundingenvironment represented in the video for at least one identifying objectindicative of an environmental characteristic; identifying, via objectrecognition, the at least one identifying object indicative of anenvironmental characteristic within the surrounding environment; andproviding information associated with the environmental characteristicto a user of the mobile device via an information address.
 2. The methodof claim 1, wherein the mobile device comprises a camera capable ofimage and/or video capture.
 3. The method of claim 1, wherein the videocomprises at least one real-world object.
 4. The method of claim 1,wherein the at least one identifying object is associated with areal-world object.
 5. The method of claim 4, wherein the real-worldobject is an advertisement.
 6. The method of claim 4, wherein thereal-world object is a consumer product.
 7. The method of claim 1,wherein the at least one identifying object indicative of anenvironmental characteristic is a cover.
 8. The method of claim 7,wherein the cover is a disc cover.
 9. The method of claim 1, wherein theat least one identifying object indicative of an environmentalcharacteristic is a goal object.
 10. The method of claim 1, wherein theat least one identifying object indicative of an environmentalcharacteristic is a candidate object.
 11. The method of claim 1, whereinthe information associated with the environmental characteristic isdependent on other information.
 12. The method of claim 11, wherein theother information comprises at least one of the following: a currenttime, an observed state of the object, and a location of the user. 13.The method of claim 1, wherein the information associated with theenvironmental characteristic is presented to the user on a display ofthe mobile device as a virtual object superimposed on the videorepresenting the surrounding environment.
 14. The method of claim 13,wherein the virtual object is interactive.
 15. The method of claim 13,wherein the virtual object provides information associated with aconsumer product.
 16. The method of claim 1, further comprisingpresenting the user with an option to purchase at least one of a productor a service.
 17. The method of claim 1, further comprising enabling atransaction based on the content information.
 18. The method of claim 1,further comprising conducting a transaction that includes the step oftransmitting purchase information from the mobile device.
 19. A mobiledevice configured to assess environmental characteristics comprising: acamera configured to capture video representing a surroundingenvironment; an image processing platform configured to: analyze thevideo to determine whether the video includes at least one indentifyingobject indicative of an environmental characteristic; identify, viaobject recognition, the at least one identifying object indicative of anenvironmental characteristic within the surrounding environment; andprovide to the user of the mobile device, via the mobile device throughan information address, information associated with the environmentalcharacteristic.
 20. The mobile device of claim 19, wherein the camera isconfigured to capture at least one of an image and the video.
 21. Themobile device of claim 19, wherein the video comprises at least onereal-world object.
 22. The mobile device of claim 19, wherein the atleast one identifying object is associated with a real-world object. 23.The mobile device of claim 22, wherein the real-world object is anadvertisement.
 24. The mobile device of claim 22, wherein the real-worldobject is a consumer product.
 25. The mobile device of claim 22, whereinthe at least one identifying object indicative of an environmentalcharacteristic is a cover.
 26. The mobile device of claim 25, whereinthe cover is a disc cover.
 27. The mobile device of claim 19, whereinthe at least one identifying object indicative of an environmentalcharacteristic is a goal object.
 28. The mobile device of claim 19,wherein the at least one identifying object indicative of anenvironmental characteristic is a candidate object.
 29. The mobiledevice of claim 19, wherein the information associated with theenvironmental characteristic is dependent on other information.
 30. Themobile device of claim 29, wherein the other information comprises atleast one of the current time, the observed state of the object, or thelocation of the user.
 31. The mobile device of claim 19, wherein theinformation associated with the environmental characteristic ispresented to the user on a display of the mobile device as a virtualobject superimposed on the video representing the surroundingenvironment.
 32. The mobile device of claim 31, wherein the virtualobject is interactive.
 33. The mobile device of claim 31, wherein thevirtual object provides information associated with a consumer product.34. The mobile device of claim 19, further comprising presenting theuser with an option to purchase at least one of a product or a service.35. The mobile device of claim 19, further comprising enabling atransaction based on the content information.
 36. The mobile device ofclaim 19, further comprising conducting a transaction that includes thestep of transmitting purchase information from the mobile device.
 37. Acomputer program product comprising non-transitory computer readablemedium storing instructions that cause a processor to execute the stepsof: capturing, via a mobile device, video representing a surroundingenvironment; analyzing, the surrounding environment represented in thevideo for at least one identifying object indicative of an environmentalcharacteristic; identifying, via object recognition, the least oneidentifying object indicative of an environmental characteristic withinthe surrounding environment; and providing information associated withthe environmental characteristic to a user of the mobile device via aninformation address.
 38. The computer program product of claim 37,wherein the mobile device comprises a camera configured to capture atleast one of an image and the video.
 39. The computer program product ofclaim 37, wherein the video comprises at least one real-world object.40. The computer program product of claim 37, wherein the at least oneidentifying object is associated with a real-world object.
 41. Thecomputer program product of claim 40, wherein the real-world object isan advertisement.
 42. The computer program product of claim 41, whereinthe real-world object is a consumer product.
 43. The computer programproduct of claim 40, wherein the at least one identifying objectindicative of an environmental characteristic is a cover
 44. Thecomputer program product of claim 43, wherein the cover is a disc cover.45. The computer program product of claim 43, wherein the target objectrepresents at least one of the following: a building, a store, avehicle, a media box, a person, a passport, a driver's license, aposter, an advertisement, on-screen content, a product, a beverage, asign, a newspaper, a magazine, a radio frequency, and a machine part.46. The computer program product of claim 37, wherein the at least oneidentifying object indicative of an environmental characteristic is agoal object.
 47. The computer program product of claim 37, furthercomprising the step of obtaining other information associated with theenvironmental characteristic.
 48. The computer program product of claim47, wherein the other information comprises at least one of the currenttime, the observed state of the object, or the location of the user. 49.The computer program product of claim 37, wherein the informationassociated with the environmental characteristic is presented to theuser on a display of the mobile device as a virtual object superimposedon the video representing the surrounding environment.
 50. The computerprogram product of claim 38, wherein the virtual object is interactive.51. The computer program product of claim 38, wherein the virtual objectprovides information associated with a consumer product.
 52. Thecomputer program product of claim 37, further comprising the step ofpresenting the user with an option to purchase at least one of a productor a service.
 53. The computer program product of claim 37, furthercomprising the step of enabling a transaction based on the contentinformation.
 54. The computer program product of claim 37, furthercomprising the step of conducting a transaction that includes the stepof transmitting purchase information from the mobile device.